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TECHNIQUE FOR SIMULATING FILM GRAIN ON ENCODED VIDEO 

CROSS REFERENCE TO RELATED APPLICATION 

5 - This application claims priority under 35 U.S.C. 110(e) to U,S, Provisional Patent 

Application Serial No. 60/162,3 8 9 filed April 11, 2003, and incorporated by reference heroin. — 

— This application claims the benefit under 35 U.S.C. § 365 of the International 
A pplication, PCT/US2004/005365, filed February 24. 2004. which was published in accordance 
10 with PCT Article 21(2) on November 4, 2004 in English and which claims the benefit of United 
States provisional patent application No. 60/462,389, filed April 10, 2003. - 

TECHNICAL FIELD 

15 This invention relates to a technique for simulating the grain of photographic film on a 

sequence of encoded video. 

BACKGROUND ART 

20 The photographic film used to make motion pictures comprises silver-halide crystals 

dispersed in an emulsion that is deposited in thin layers on a film base. The exposure and 
development of these crystals form the photographic image, which is made at the largest scale of 
discrete particles of silver. With color negatives, tiny blobs of dye exist on the sites where the 
silver crystals form after chemical removal of the silver following development. These small 

25 specs of dye form the 'grain' in color film. Grain occurs randomly in the resulting image because 
of the random formation of silver crystals on the original emulsion. Within a uniformly exposed 
area, some crystals develop by exposure, others not. Grain varies in size and shape. The faster 
the film (i.e., the greater the sensitivity to light), the larger the clumps of silver formed and blobs 
of dye generated, and the more they tend to group together in random patterns. The grain pattern 

30 is typically known as 'granularity'. 

The naked eye cannot distinguish individual grains, which vary from 0.0002 mm to 0.002 
mm. Instead, the eye resolves groups of grains, which the viewer identifies as film grain. The 
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larger the image resolution, the more likely the viewer will perceive the film grain. While clearly 
noticeable in cinema and high-definition images, film grain progressively loses prominence in 
Standard Definition Television (SDTV) images and becomes imperceptible in even smaller 
formats. 

5 Typically, within the domain of video coding, efforts currently exist to improve the 

performance of the encoder to encode film grain at high bit-rates. Note that, since film grain 
becomes noticeable only in large image formats such as Standard Definition Television (SDTV), 
High Definition Television (HDTV) and above, the problem of film grain affects mainly 
professional encoders. 

10 Present-day studies about film grain have targeted mainly photographic imaging 

applications (image edition, medical imaging, satellite imagery, astronomy and 
astrophotography, etc.). There currently exist software applications in the market place (Adobe 
After Effects, Photoshop, etc.) that provide solutions to this problem. In the domain of video 
coding, studies have addressed the desirability of removing film grain with the goal of improving 

15 the coding performance at mid and low bit-rates. However, the literature contains no specific 

strategies to encode film grain differently from other high frequency information, such as texture 
or contours. 

In some sense, film grain constitutes the correlated noise inherent in the physical process 
of developing of photographic film. The presence of film grain denotes 'real-world* images 

20 contrast to 'computer-generated' material with no film grain. Images without film grain look like 
hyper-reality, simply too flat and too sharp to be real. For this reason, film grain remains a 
desired 'artifact' in most images and video material. However, because film grain arises from a 
random process affecting high frequencies, the encoding process typically affects film grain. 
Lossy encoders conmionly achieve part of their compression gain by avoiding the transmission 

25 of the high frequencies in the DCT domain. Such frequencies are typically associated with noise, 
sharp edges and texture, but also with the film grain. Even at high bit-rates, film grain becomes 
difficult to preserve, and always at a high compression cost. 

Thus, a need exists to alleviate the cost of encoding film grain without degrading the 
perceived quality of the displayed image(s). 



30 
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BRIEF SUMMARY OF THE INVENTION 

Briefly, in accordance with present principles, there is provided a method for simulating 
5 film grain in an encoded image. The provided method entails film grain extraction and film grain 
characterization at the encoder to enable the film grain restoration at the decoder. In a preferred 
embodiment, the method commences by identifying the film grain that is present in an incoming 
image. Thereafter, the film grain is characterized based either on the type of film, or using a 
particular model. The incoming image is then encoded and the film grain characterization 
10 information transmitted which allows a decoder decoding the encoded image to restore the film 
grain in accordance with the information transmitted. The method transmits the film grain 
particulars as parallel information to the video coded stream, typically as a film grain 
Supplemental Enhancement Information (SEI) message when using the ITU-T H.264 video 
coding standard. 

15 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIGURE 1 illustrates a block diagram of a first embodiment of a system in accordance 
with the present principles for simulating film grain in accordance with the present principles; 
20 FIGURE 2 illustrates a block diagram of a second embodiment of a system in accordance 

with the present principles for simulating film grain in accordance with the present principles; 
and 

FIGURE 3 illustrates a block diagram of a third embodiment of a system in accordance 
with the present principles for simulating film grain in accordance with the present principles. 

25 

DETAILED DESCRIPTION 

Different types of motion picture film have different film grain pattems well known in the 
art. Thus, identifying the type of film stock used to record the original source material allows 
30 identification of the film grain of the encoded material. Table 1 lists an individual numerical 
identifier for each of a plurality of exemplary film stocks, each having a known film grain 
pattern. 
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Table 1 



Identifier 


Film grain model 


0 


Kodak Vision 200T 5274 


1 


Kodak Vision 259D 5246 


2 


Kodak Vision 320T 5277 


3 


Kodak Vision SOOT 5279 






N 


Kodak PROFESSIONAL SUPRA 100 



Knowing the type of film originally used to record the pictures embodied in an encoded 
video assures optimal quality when restoring the film grain following image decoding. 
5 Additional information concerning down-sampling factors that apply to the original image also 
will prove helpful the film grain restoration at the correct scale. Note that Table I is not intended 
to be all inclusive but merely exemplary. Other film stocks from other film manufacturers will 
have separate identifiers. 

In the absence of any knowledge of the type of film stock used to record the original 

10 source, film grain characterization can occur through a modeling process. The modeling process 
seeks to reduce the amount of film grain characterization information to be transmitted by 
providing a compact representation of the film grain pattern and intensity. Such an approach 
provides an estimate of the original film grain, which can differ from the actual film depending 
on the selected modeling process. When the system that models the film grain at an encoder is 

15 enabled to select among more than one modeling methods to characterize the film grain of the 
incoming images, a decoder should receive at least some information identifying the modeling 
method that was selected. In a particular embodiment, the modeling process could provide a 
compact representation of the film grain according to a non-parametric model. In another 
embodiment, the modeling process could consist in a parameterization process according to a 

20 pre-defined mathematical model. To illustrate this last embodiment. Table 2 provides an 
example of several different mathematical models could be used to describe film grain. 



TABLE 2 



Identifier 


Film grain model 


0 


f(x,y,c) = d* n 


1 


f(x,y,c) = s(x,y,c) + k*s(x,y,c) + d* n 
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2 


f(x,y,c) = a * f(x-l,y-l,c) + b*f(x,y,c-l) + d*n 


3 


f(x,y,c) = a * r(x,y,c) - b*s(x,y,c) + d*n 






N 


f(x,y,c) = a*[ d(x-l,y,c) + f(x,y-l,c)] + b*f(x,y,c-l) + d*n 



The use of parametric models requires the transmission of the estimated set of parameters. 
The parameters will depend on the type of model as specified in Table 2, or in the simplest case, 
will correspond to a unique film grain model known a priori from the type of film as described in 
Table 1. The parameters of a given film grain model should allow adjustment of the size of the 
film grain, its intensity, its spatial correlation, its color correlation, etc. As an example, assume 
the following formula serves to model the film grain in an image: 



f(x,y,c) = a*[ f(x-l,y,c) + f(x,y-l,c)] + b*f(x,y,c-l) + d*n 



where f(x,y,c) represents the film grain of the pixel at coordinates (x,y) on the color component c, 
and n represents a Gaussian noise of mean zero and variance one. According to this model, an 
encoder should transmit the parameters 'a\ 'b' and 'd* to allow a decoder to simulate the original 
film grain. Note that the parameters of the model could depend on other factors, such as signal 
intensity, the color component, etc. Hence, transmission of the film grain model parameters 
actually entails transmission of sets of model parameters for each different case. 

FIGURE 1 depicts a block schematic diagram of a first embodiment of a system 10 in 
accordance with the present principles for performing film grain simulation. The system 10 
includes a Film Grain Remover 22 that serves to remove the film grain from an input video 
stream 12 to yield a filtered video stream 24 received at a Video Encoder 13. Film grain removal 
constitutes a particular case of noise filtering where the noise signal appears correlated with the 
image signal. Thus, the Film Grain Remover 22 can take the form of a classical image filter, 
although such a filter will not necessarily provide optimal performance. The Video Encoder 13 
encodes the filtered video stream 24 to yield a coded video stream 14 for receipt at a Video 
Decoder 15 that decodes the coded stream to yield a decoded video stream 16. The Video 
Encoder 13 and the Video Decoder 15 utilize the same video coding scheme as are well known in 
the art. For example, the video coding scheme could comprise the ITU-T H.264 video-coding 
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standard, or another type of block-based coding. Encoders and decoders that utilize the MPEG-2 
and the ITU-T H.264 standard are well known. 

The system 10 also includes a Film Grain Characterizer 23 that receives the input video 
stream 12 and the filtered video 24 stream. From these video streams, the Film Grain 
5 Characterizer 23 outputs film grain characterization information 25. In its simplest 

implementation, the Film Grain Characterizer 23 characterizes the grain in the input video 12 
from a limited set of film grain samples. As an example, film grain samples can be obtained by 
subtracting the filtered video 24 from the input video 12. In another embodiment, the Film Grain 
Characterizer 23 can take the form of a look up table that outputs film grain characterization 

10 information in accordance with the type of film stock originally used to record the picture 

embodied in the input video stream 12. Thus, for example, metadata accompanying the input 
video stream 12 could identify the type of film stock in accordance with Table 1. Using its look- 
up table, the Film Grain Characterizer 23 will provide the parameters for the identified film, as 
well as corresponding film grain model for that film. In another embodiment, the Film Grain 

15 Characterizer 23 can make use of a film grain modeling process that characterizes the film grain 
in the input video stream 13 by a set of parameters estimated according to a pre-defined method. 

Following the film grain characterization, a Film Grain characterization information 
Encoder 26 transmits an encoded information stream containing the film grain characterization 
information to a Film Grain characterization information Decoder 28 in parallel to the encoded 

20 video stream 14 transmitted by the Video Encoder 13 to a the Video Decoder 15. Both the Video 
Encoder 13 and the Film Grain characterization information Encoder 26 use the same encoding 
scheme. Thus, for example, when the Encoder 26 utilizes the ITU-T H.264 video-coding 
standard for encoding, the coded film grain characterization information stream 27 can take the 
form of the film grain Supplemental Enhancement Information (SEI) message as defined in the 

25 ITU-T H.264 video coding standard. 

The Film Grain characterization information Decoder 28 decodes the coded film grain 
characterization information stream 27 to yield a decoded film grain characterization information 
stream 29 for input to a Film Grain Restoration Processor 30. In its simplest form, the Film 
Grain Restoration Processor 30 takes the form of a pattern adder that blends a film grain pattern 

30 with the decoded video stream 16 supplied by the Video Decoder 15 to yield a decoded video 
stream with simulated film grain 31. In another embodiment, the Film Grain Restoration 
processor 30 can comprise a pattern generation processor that creates a film grain pattern 
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according to the film grain characterization information in the decoded film grain 
characterization information stream 29. 

Notice that the film grain characterization information can vary dynamically through a 
video sequence. Thus, different groups of frames can require the transmission of different film 
5 grain characterization information. In this way, the Film Grain Restoration processor 30 can 
update the film grain pattern depending on the transmitted characterization information. 

FIGURE 2 depicts a second embodiment 10' of a system for simulating film grain in 
accordance with the present principles. The system 10* shares many of the same elements as the 
system 10 of FIG. 1 and like reference numbers describe like elements. Indeed, the system 10' of 

10 FIG. 2 differs only in the absence of the Film Grain characterization information Encoder 26 and 
Film Grain characterization information Decoder 28 of FIG. 1. The system 10' of FIG. 2 uses 
the Video Encoder 13 and Video Decoder 15 to encode and decode respectively the film grain 
characterization information 25 output of the Film Grain Characterizer 23. The system 10' of 
FIG. 2 requires the use of a video coding standard that supports the transmission film grain 

15 characterization information as parallel enhancement information. Thus, for example, when the 
Video Encoder 13 utilizes the ITU-T H.264 video-coding standard for encoding, the coded film 
grain characterization information stream 27 can take the form of a Supplemental Enhancement 
Information (SEI) message as defined in the ITU-T H.264 video coding standard. 

FIGURE 3 depicts a third embodiment 10" of a system for simulating film grain in 

20 accordance with the present principles. The system 10" shares many of the same elements as the 
system 10* of FIG. 2 and like reference numbers describe like elements. Indeed, the system 10" 
of FIG. 3 differs only in the absence of the Film Grain Remover 22 of FIG. 2. The system 10" 
of FIG. 3 uses the reconstructed images available at the Video Encoder 13 to simulate the result 
of removing film grain. The system 10" of FIG. 3 affords two advantages as compared to the 

25 systems 10 of FIG. 1 and 10' of FIG 2. First, the system 10' ' of FIG. 3 reduces the computational 
complexity related to film grain removal, and secondly, it adapts the film grain characterization 
to the amount of film grain suppressed by the Video Encoder 13. Once the Film Grain 
Characterizer of FIG. 3 disposes of both the input video 12 with film grain, and a reconstructed 
video 24 resulting from Video Encoder 13, it can accomplish the task of characterizing the 

30 observed film grain. 

In some cases film grain characterization can involve color conversion and/or pixel 
sample interpolation depending on the original file format. For high quality applications, film 
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grain modeling occurs in the RGB color space, which better approximates the layer configuration 
of the physical process of film developing. The simplest parametric model can assume the film 
grain to be a Gaussian noise of zero mean uncorrelated with the image signal. In this case, only 
the transmission of the standard deviation of the Gaussian function is required. More 
complicated models can require the transmission of different parameters for each color 
component and/or for different gray level sets. The choice of a model can be strongly related to 
the affordable complexity at the decoder side, the number of bits available for encoding the SEX 
message and mainly the desired quality on display. 

In accordance with the present principles, film grain encoding is performed by the step of 
(a) characterizing the film grain at an encoder, (b) transmitting the film grain characterization 
information in parallel to the coded video stream and (c) blending the simulated film grain with 
the decoded video. As discussed, film grain simulation can rely on a predefined model, which 
reproduces the film grain of a specific type of film, or can occur by parameterization using a 
mathematic model. In all cases, film grain restoration occurs prior to display. Images with 
added film grain are never used within the decoding process; however, some parallelization could 
be possible for causal models. 

The foregoing describes a technique for encoding film grain in a video image. 



